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Abstract 

The absence of an economy of scale in spaceflight hardware makes additive manufacturing an 
immensely attractive option for propulsion components. As additive manufacturing techniques 
are increasingly adopted by government and industry to produce propulsion hardware in human- 
rated systems, significant development efforts are needed to establish these methods as reliable 
alternatives to conventional subtractive manufacturing. One of the critical challenges facing 
powder bed fusion techniques in this application is variability between machines used to perform 
builds. Even with implementation of robust process controls, it is possible for two machines 
operating at identical parameters with equivalent base materials to produce specimens with 
slightly different material properties. The machine variability study presented here evaluates 60 
specimens of identical geometry built using the same parameters. 30 samples were produced on 
machine 1 (Ml) and the other 30 samples were built on machine 2 (M2). Each of the 30-sample 
sets were further subdivided into three subsets (with 10 specimens in each subset) to assess the 
effect of progressive heat treatment on machine variability. The three categories for post- 
processing were: stress relief, stress relief followed by hot isostatic press (HIP), and stress relief 
followed by HIP followed by heat treatment per AMS 5664. Each specimen (a round, smooth 
tensile) was mechanically tested per ASTM E8. Two formal statistical techniques, hypothesis 
testing for equivalency of means and one-way analysis of variance (ANOVA), were applied to 
characterize the impact of machine variability and heat treatment on six material properties: 
tensile stress, yield stress, modulus of elasticity, fracture elongation, and reduction of area. This 
work represents the type of development effort that is critical as NASA, academia, and the 
industrial base work collaboratively to establish a path to certification for additively 
manufactured parts. For future flight programs, NASA and its commercial partners will procure 
parts from vendors who will use a diverse range of machines to produce parts and, as such, it is 
essential that the AM community develop a sound understanding of the degree to which machine 
variability impacts material properties. 
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Introduction 


Additive manufacturing (AM) encompasses a broad swath of technologies that consist of 
“joining materials to make objects from three-dimensional model data, usually layer by layer, as 
opposed to subtractive manufacturing methodologies” [1]. AM first emerged in the 1980s with 
stereolithography and has since grown to include fused deposition modeling (FDM), selective 
laser melting (SLM), selective laser sintering (SLS), electron beam melting (EBM), laminated 
object modeling (LOM), and laser engineered net shaping (LENS) [2]. While there is no formal 
recognized classification schema for AM, processes can be broadly categorized according to 1) 
the technique used to deposit the layers, 2) the energy source/method by which the deposited 
layers are fused together, 3) the state of the starting materials (liquid, wire, filament, powder, 
etc.) and/or 4) the type of material (biomaterial, polymer, metal, composite, ceramic) [1]. The 
level of activity in AM research has burgeoned in recent years as the field has become a national 
priority in science and technology policy. The potential of AM to benefit multiple sectors, return 
manufacturing expertise to US soil, and promote future industry has been recognized by the 
Obama administration, the Office of Management and Budget, and the Office of Science and 
Technology Policy [3], Currently, government directed initiatives such as America Makes and 
the Materials Genome Initiative (MGI) work to facilitate collaboration in materials research and 
transfer knowledge among industry, academia, and government agencies as well as develop data 
management tools (such as informatics databases) that will accelerate the development (and 
subsequent adoption by industry) of new materials and manufacturing techniques [4]. AM 
represents a large focus of these efforts. 

NASA Marshall Space Flight Center in Huntsville, Alabama has been at the forefront of 
additive manufacturing research since 1991, when the additive manufacturing laboratory was 
established with two primary purposes: 1) to provide a rapid prototyping capability for 
spaceflight hardware and 2) develop methods to build parts in-space using material extrusion 
techniques such as FDM. In subsequent years, the laboratory became part of a 
government/industry consortium - the National Center for Manufacturing Science (NCMS) — 
with an emphasis on the development of metallic additive manufacturing systems. In recent 
years, the laboratory’s has focused on maturing metallic AM techniques to produce rocket 
propulsion hardware, specifically the qualification of electron beam and powder bed fusion 
processes for this application. 

As two decades of innovation in materials and processes have transitioned AM from a 
laboratory curiosity to a manufacturing technique capable of fabricating production quality parts, 
the aerospace and aviation sector has begun to adopt AM and integrate it into their design 
processes. The benefits of AM for propulsion components, which generally have long lead times 
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and large costs when produced by traditional subtractive methods, is widely recognized. For 
instance, an additively manufactured gas generator duct for NASA’s J-2X engine imparted a 
70% reduction in cost and a 50% reduction in schedule/manufacturing time over its 
conventionally manufactured counterpart [5]. For the flex joint on the RS-25 engine, the 
selective laser melting (SLM) process was able to reduce the part count from 45 (heritage 
design) to 17 and the number of welds from 70 to 26 [5]. These examples illustrate that AM is a 
particularly attractive option for the low-rate production, complex hardware geometries that are 
characteristic of propulsion hardware. 

While industry and NASA are eager to capitalize on AM’s ability to reduce part counts, 
minimize the number of welds and brazes, and impart schedule and costs savings, no new 
technology is adopted without challenges. There are four key technical hurdles that NASA and 
the broader aerospace sector must address as we work together to develop and certify AM parts 
and processes for use in spaceflight (and particularly human-rated spaceflight) systems [6]: 

1) Materials characterization - understanding the influence of AM build parameters, 
powder characteristics, thermal processing, and surface texture on the material 
properties of the end-use part 

2) Standard design practices - development of guidelines for mechanical design, testing 
specifications, management of uncertainties associated with material properties and 
operational environments 

3) Process modeling, monitoring, and control - physics based modeling of transport 
phenomena in AM processes, in situ defect detection, prediction of material 
properties based on material characteristics and build parameters 

4) Establishing flight certification logic - development of a part classification system 
using a risk-based approach, understanding of part failure modes and establishment 
of NDE and fracture control requirements, approaches to lot acceptance testing 

The additive manufacturing laboratory at NASA Marshall and its partners have significant 
ongoing activities in all of these areas. The study presented in this paper focuses on 
characterizing the effect of machine variability on additively manufactured materials. This work 
is most closely tied to area (1) (materials characterization and understanding of process/property 
relationships), but is also key to standard design practices (since designers and analysts need to 
know the degree of variability in material properties, specifically the minimum characteristic 
material property values used for design, that can be expected when identical parts are produced 
on different AM machines), process modeling and control (as a smart developer and procurer of 
AM parts, we need to understand what factors contribute to machine variability effects and the 
procedures/controls that most effectively minimize variation in properties among end-use parts), 
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and flight certification (property variability, whether a consequence of machine differences or 
other process factors, influences the selection and development of procedures for lot acceptance 
testing). 


Process Variability in AM 

The design flexibility and cost/schedule savings enabled by AM make this class of 
technologies particularly attractive for production of aerospace components. However, the use 
of AM processes for production of aerospace grade materials rather than prototyping is still 
regarded as nascent. One of the central challenges of AM implementation in industry is to 
identify, quantify, and control the variables that can influence the material properties of the end- 
use part. Designers and analysts rely on statistically based minimum material properties to 
establish safe margins and ensure that a part will be able operate in its intended use environment. 
These properties are usually characterized through a material allowables program and 
subsequently published in materials handbooks such as MMPDS (Metallic Materials Properties 
Development and Standardization) and CMH-17 (Composite Materials Handbook 17). As an 
emerging technology, AM materials and process (M&P) specifications do not exist to the same 
level of fidelity and control as those established in MMPDS, CMH-17, and similar governing 
M&P documents recognized by government and industry for conventionally manufactured 
materials [7]. For aerospace, in many cases AM parts are regarded as process specific and 
certification activities are undertaken on a part by part basis. 

In lieu of standards, organizations who use AM in production and/or critical hardware 
may undertake their own small-scale allowable programs to explore processing conditions and 
characterize the effect of process variables on material properties with the ultimate goal of using 
this knowledge to establish minimum property design values that serve as critical inputs to 
designers and analysts. In addition to a reduced sample size compared to the recommendations of 
MMPDS and CMH-17, another limitation of these programs is that (in most cases) specimens or 
parts are produced on a single AM machine. Previous experience has demonstrated that 
variation in material properties may not only be sensitive to process variables, but specific to the 
machine the specimens were produced on. There is a concern from a process control standpoint 
that design minimum properties might be slightly different when AM parts are produced on 
different machines, even if the build parameters, materials, thermal processing conditions, etc. 
remain the same. 
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There is little in the published literature assessing the machine specificity of AM parts. 
While there is some speculation that powder bed fusion processes may be more susceptible to 
machine variability effects than AM processes that are extrusion based, there are no formal 
studies to substantiate this. Based on the MSFC laboratory’s experience with powder bed fusion 
techniques, it is possible for two machines operating at identical parameters with equivalent base 
materials to produce specimens with slightly different material properties. Quantifying the 
impact of machine variability on material properties and developing process controls to minimize 
these effects are key to establishing AM methods as reliable alternatives to traditional subtractive 
manufacturing [8]. Understandably, there is some resistance among the standards community 
toward the inclusion of emerging technologies such as AM in design handbooks due to process 
variability — manufacturing processes must demonstrate that they are highly repeatable, reliable 
and consistent before their incorporation into standards documents can be considered. An 
understanding of machine variability in AM will accelerate the technology’s commercial 
acceptance and create a knowledge base that is critical to transitioning AM from the custom LRP 
(low rate production) sphere it currently occupies to the world of mass HRP (high rate 
production), where production parts are fabricated at a rapid rate with confidence that they will 
meet or exceed all stringent functional requirements for spaceflight hardware. 

Characterization of machine variability is critical to NASA, academia, and the industrial 
base as we collaborate to establish a path to certification for additively manufactured hardware. 
For future flight programs, NASA and its commercial partners will procure AM components 
from vendors who use a diverse range of machines and AM equipment to produce parts. A 
sound understanding of the degree to which machine variability impacts material properties is 
essential for lot acceptance and hardware certification. 

Methodology 

In an effort to address the knowledge gap related to the effect of machine variability on 
material properties, a small-scale study on this subject was undertaken at NASA Marshall Space 
Flight Center. The advanced manufacturing team at NASA MSFC operates two Concept Laser 
Selective Laser Melting (SLM) machines. Although the material group for this investigation 
considered only the nickel-based alloy Inconel 718, these machines are also capable of 
processing high grade steels, Aluminum alloys, Titanium alloys, and pure Titanium. The Ml 
machine is best-suited for the production of small to medium sized components. The build 
envelope for the Ml is 250 x 250 x 250 mm (x,y,z) and the laser system is a fibre laser capable 
of providing up to 400 W of energy. The M2 from Concept Laser is also a 400 W laser system 
but has a slightly larger build box (250 x 250 x 280 mm). The Ml and M2 were used to produce 
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60 tensile specimens (30 from each machine) at identical default machine parameters using 
virgin Inconel 718 powder from the same lot. 


The study had two major objectives: 

1) To determine the amount of variability in mechanical material properties between the 
two machines 

2) To determine the effect of heat treatment on machine variability 


The 60 Inconel 718 round tensile specimens were tensile tested in accordance with ASTM E8, 
“Standard Test Methods for Tension Testing of Metallic Materials” [9]. The specimens were 
built in the vertical (Z) orientation. The machined specimens had a nominal diameter of 0.250 
inches and a nominal length of 3.0 inches. The environment for the testing was lab air at room 
temperature, approximately 70 degrees Fahrenheit. Tests were conducted in displacement 
control at an initial rate of 0.02 in/min. This rate was maintained up to 2% strain, after which the 
rate was increased to the final displacement rate of 0.05 in/min. Stress data were obtained from 
load measurements from a calibrated 20,000 pound load cell and pre-test specimen dimension 
measurements. Strain data were obtained from displacement measurements from a 1.00 inch 
extensometer calibrated to 50% strain. Ultimate stress, yield stress, and fracture elongation 
values were determined from the resulting stress-strain curve according to procedures outlined in 
ASTM 38. Elastic modulus was reported for reference, but specific modulus tests are needed for 
true values. The following material properties were reported for each of the sixty specimens: 
Ultimate tensile strength outs, yield strength cy, elastic modulus E, fracture elongation e 
(expressed as a percent), and reduction in area Ar (also a percent). 

All specimens were stress relieved at 1950 F +/- 25 F for 1.5 hours, -5/+15 min., followed by a 
2-4 Bar argon quench. The stress relief process is necessary to separate the samples from the 
build plate. Forty specimens (20 from each machine) underwent a hot isostatic press (HIP) 
treatment at 2050 +/- 25 F and 15,000 +/- 250 psi for 3 hrs. +/- 15 min. Twenty of the HIP 
specimens (10 from each machine) were heat treated according to AMS 5664E, which consists 
of a solution heat treatment at 1900-1950 F +/- 25 for a time commensurate with cross-sectional 
thickness, and cooling at a rate equivalent to an air cool or faster, followed by a precipitation 
heat treatment at 1400 F +/- 15 F until a total precipitation hat treatment time of 20 hrs. has been 
reached, then cooled. 

To facilitate comparative analysis, the data were grouped according to machine and 
postprocessing conditions. The group name, number of specimens in the group, the SFM 
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machine (or machines) used to build the specimens, and the post-build conditioning that 
correspond to the data groups are summarized in Table 1.* The machine used to produce the 
specimens and the post-build conditioning for a group can be identified from the group name. 
For instance. Ml SR+HIP refers to the group of specimens that were produced on the Concept 
Laser Ml and underwent stress relief followed by hot isostatic press. 


Table 1 Summary of specimen groups 


Specimen group name 

Number of 
specimens 

SLM machine(s) 

Post-build conditioning 

Ml SR 

10 

Ml 

SR 

M2 SR 

10 

M2 

SR 

Ml SR+HIP 

10 

Ml 

SR+HIP 

M2 SR+HIP 

10 

M2 

SR+HIP 

Ml SR+HIP+HT 

10 

Ml 

SR+HIP+HT 

M2 SR+HIP+HT 

10 

M2 

SR+HIP+HT 


While Table 1 represents the groupings that are the most useful for direct comparative analysis 
of material properties, we can also construct pooled data groups. These groupings are comprised 
of one of the following: a) samples which were subject to the same postprocessing conditions but 
were built on different machines or b) specimens built on the same machine but which 
underwent different postprocessing regimes. The entire set of 60 samples are analyzed in the 
M1+M2 data group. The pooled data groupings appear in Table 2. 


*Acronyms are used to designate post-build conditioning. SR indicates the specimens underwent stress relief, HIP 
corresponds to hot isostatic press, and HT is heat treatment (AMS 5664E). For example, a specimen group 
identified by SR+HIP+HT indicates that the specimens were stress relieved, underwent HIP, and then heat treatment 
per AMS 5664E. 
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Table 2 Summary of pooled specimen groups. 


Specimen group name 

Number of 
specimens 

SLM machine(s) 

Post-build conditioning 

M1+M2 

60 

Ml, M2 

Various 

Ml 

30 

Ml 

Various 

M2 

30 

M2 

Various 

SR 

20 

Ml, M2 

SR 

SR+HIP 

20 

Ml, M2 

SR+HIP 

SR+HIP+HT 

20 

Ml, M2 

SR+HIP+HT 


The analyses of the data groupings are summarized in the following sections. Single 
variable analyzes characterize the material properties for a single data grouping. The two-sample 
comparisons analyze data originating from different groupings, making visual comparisons and 
testing hypotheses to determine if differences in properties between the groups are statistically 
significant. The ANOVA analysis compares material properties for multiple groups. 


Single Group Analyses 

Raw data from the mechanical test reports were used to calculate the mean values of ultimate 
tensile strength outs, yield strength ay, elastic modulus E, fracture elongation e, and reduction in 
area Ar for each data grouping. Table 3 summarizes the mean material property values for the 
groupings. 
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Table 3 Mean material property data for specimen groups 


Grouping 

Sample 

size 

Ultimate 

tensile 

stress 

(KSI) 

Yield 

stress 

(KSI) 

Modulus 

(MSI) 

Elongation 

(%) 

Reduction 
in area (%) 

M1+M2 

60 

184.4 

143.1 

27.7 

25.7 

33.9 

Ml SR 

10 

164.7 

128.3 

22.4 

21.2 

25.3 

M2 SR 

10 

171.9 

129.6 

27.4 

30.8 

42.1 

Ml SR+HIP 

10 

182.8 

132.6 

28.7 

23.0 

25.2 

M2 SR+HIP 

10 

186.0 

134.3 

29.5 

28.4 

37.3 

Ml 

SR+HIP+HT 

10 

199.6 

166.2 

29.1 

24.8 

35.8 

M2 

SR+HIP+HT 

10 

201.7 

168.0 

29.0 

25.8 

38.0 

Ml 

30 

182.4 

142.3 

26.7 

23.0 

28.7 

M2 

30 

186.5 

144.0 

28.6 

28.3 

39.1 

SR 

20 

168.3 

129.0 

24.9 

26.0 

33.7 

SR+HIP 

20 

184.4 

133.5 

29.1 

25.7 

31.2 

SR+HIP+HT 

20 

200.6 

167.1 

29.1 

25.3 

36.9 


The scatterplots in Figures 1 through 5 graphically illustrate the differences between the group 
means for tensile strength outs, yield strength oy, elastic modulus E, fracture elongation e, and 
reduction in area Ar, respectively. From these plots, some trends start to emerge: 1) that identical 
specimens produced on different machines do seem to have slightly different properties 
(particularly for outs, oy, and E), 2) that the differences in mean material properties for 
equivalent specimens built on different machines seem to decrease with subsequent 
postprocessing (HIP and heat treatment), and 3) that HIP and HT generally improve mechanical 
properties. Further statistical tests (summarized in the two group comparison and multigroup 
analyses sections) assess whether the discrepancies in means between groups are statistically 
significant and if the apparent equalization effect of HIP and heat treatment is real. 
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Mean ultimate tensile stress 
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Figure 1 Comparison of mean ultimate tensile stress for data groupings 
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Mean yield stress 
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Figure 2 Comparison of mean yield stress for data groupings 
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Mean elastic modulus 
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Figure 3 Comparison of mean elastic modulus for data groupings 
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Mean Fracture Elongation 
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Figure 4 Comparison of mean fracture elongation for data groupings 
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Mean Reduction of Area 
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Figure 5 Comparison of mean reduction of area for data groupings 


Two Group Comparisons 

Two-variable statistical comparative procedures were used to compare the means, medians, and 
standard deviations for the material properties (ultimate tensile stress a u , yield stress a y , 
modulus E, % elongation 5, and % reduction in area A ) between selected data. The two-group 
comparisons and ANOVA (next section) are complementary to the single group analyses. The 
latter asked the question “what are the differences in material property values for the various data 
groupings?” The goal of the two variable analyses and ANOVA is to determine whether these 
differences are statistically significant. 
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Analyses were restricted to comparisons that would help answer two key questions of interest in 
this study: 


1) Is there a statistically significant difference in material properties for samples which 
underwent the same heat treatment but were produced on different machines? 

2) Is there a statistically significant difference in material properties for samples which were 
produced on the same machine but underwent different heat treatments/postprocessing 
regimes? 


With these points in mind, the following data groups were compared: Ml SR vs. M2 SR, Ml 
SR+HIP vs. M2 SR+HIP, Ml SR+HIP+HT vs. M2 SR+HIP+HT, Ml SR vs. Ml SR+HIP, Ml 
SR vs. Ml SR+HIP+HT, Ml SR+HIP vs. Ml SR+HIP+HT, M2 SR vs. M2 SR+HIP, M2 SR vs. 
M2 SR+HIP+HT, and M2 SR+HIP vs. M2 SR+HIP+HT. 

Analyses were additionally limited to comparison of groups with equal sample sizes. While a 
large difference in sample size does not invalidate the statistical tests used to make a comparison 
(normality is the relatively more important underlying assumption), the disparity does create 
skewness in the t-statistic used to detennine whether the difference in means for the groups is 
statistically significant. We must be particularly cautious of the case where the larger variance is 
associated with the smaller sample size, since the t-statistic will then be dominated by the 
variance for the smaller sample [10]. 

The most valuable comparisons are those which satisfy the following criteria: 

a) sample sizes are equal 

b) only one variable is different for the data sets (ex. they were produced on the same machine 
but one set underwent an additional heat treatment or the specimens were subject to the same 
postprocessing conditions but produced on different machines) 

c) the assumption for normality are not violated (kurtosis and skewness metrics are between -2 
and 2) 
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Summary of statistical tests 

• Comparison of means 

The first fonnal comparison between the two data sets is to test the hypothesis that the means 
of the populations from which the data arise are equal (the null hypothesis) versus the 
alternative hypothesis that they are different. The procedure used to compare the means is 
the t-test (performed here at the 95% confidence level). If the p-value associated with the 
test is less than 0.05, there is sufficient evidence to reject the null hypothesis that the means 
are equal and declare the population means to be statistically different. 

• Comparison of variances 

We are not only interested in comparing the mean material properties for the specimens, but 
also want to catalog shifts and differences in variability. The F-test for equivalence of 
variances is used to detect such differences and determine their statistical significance. The 
F-statistic is used to test the hypothesis that the variance of the populations from which the 
two data sets arise are equal. If the p-value associated with the test is less than 0.05, there is 
sufficient evidence to reject the null hypothesis that the variances are equal and declare the 
population variances to be statistically different. 

• Comparison of medians 

While the mean is the statistical metric most frequently used in characterization and 
summary of data, its primary drawback is its sensitivity to outliers in the data set which have 
a tendency to skew the average to a higher or lower value (and, depending on the number and 
degree of the outliers, may result in a value that is not truly representative of the data). The 
median, in contrast, is an unbiased estimator that will not be affected by the presence of 
outliers (an outlier is defined as a point that lies beyond 3 standard deviations of the mean). 

The Mann-Whitney test for equivalence of means is similar to the t-test. A p-value less than 
0.05 leads us to conclude that the medians of the two populations from which the samples 
originate are significantly different [11]. 

• Comparison of distributions 

The Kolmogorov-Smirnoff test compares the maximum vertical distance between the 
cumulative probability distribution for two samples to assess whether they are likely to have 
originated from significantly different populations [11]. Asa nonparametric test, the K-S test 
does not require an assumption of normality and can thus be applied to the pooled data sets in 
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the variability study (which have non-normal characteristics) without reservation. The K-S 
test is also less sensitive to small sample sizes and/or disparities in sample size between data 
sets. 

The p-value associated with the K-S statistic is used to determine whether the population 
distributions associated with the samples being compared are significantly different (yet 
makes no assumptions about the shape, symmetry, or parameters of the distributions). A p- 
value of less than 0.05 leads to the conclusion that there is a significant difference at the 95% 
confidence level. Conversely, a p-value of greater than 0.05 means that there is not enough 
evidence to reject the null hypothesis that the samples originated from the same population 
distribution. A visual tool for interpreting the results of the K-S test is the quantile plot. 


Discussion and conclusions (two group comparisons) 

Key observations from the two variable analyses that appear consistently are: 

1) Identical specimens built at the same parameters and subject to the some postprocessing 
and heat treatment protocols but produced on different machines can have material 
properties that are different at a statistically significant level. This is demonstrated by the 
group comparisons Ml SR/M2 SR and Ml SR+HIP/M2 SR+HIP, for which the mean, 
median, standard deviation, and population distribution are typically different for each of 
the five properties evaluated (although the HIP step seems to erode the differences in 
modulus). This is not the case for the Ml SR+HIP+HT/M2 SR+HIP+HT comparison 
(see point 3). Table 4 tabulates the properties which were determined to be different (at 
the 0.05 level) for each statistical metric in the Ml SR/M2 SR and Ml SR+HIP/M2 
SR+HIP comparisons. Table 5 indicates the average percent difference in each property 
for the two groups. 

Table 4 Statistically Significant Differences (p<.05) Detected in Group Comparisons 


Comparison 

Means 

Variance 

Median 

Distribution 

Ml SR vs M2 
SR 

Gu, Gy, E , £, Ar 

Gu, £, Ar 

Ou, Oy, E, e, Ar 

Ou, Oy, E, 6 , Ar 

Ml SR+HIP vs 
M2 SR+HIP 

Ou, Oy, 6 , Ar 

Gu, £, Ar 

Ou, Oy, 6 , Ar 

Ou, Oy, 6 , Ar 
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Table 5 Summary of Percent Difference in Mean Material Properties for Group 
Comparisons 


Comparison 

Gu 

Gy 

E 

e 

Hr 

Ml SR vs M2 
SR 

4.3 

1.0 

22.0 

45.4 

66.8 

Ml SR+HIP 
vs M2 
SR+HIP 

1.8 

1.3 

2.6 

23.2 

48.4 


2) For groups of identical specimens produced on the same machine but where one group is 
subject to additional postprocessing (HIP) or heat treatment, the general effect of these 
processes is a statistically significant improvement in material properties. This 
conclusion is illustrated by the following comparisons: Ml SR vs Ml SR+HIP, Ml 
SR+HIP vs Ml SR+HIP+HT, M2 SR vs M2 SR+HIP, M2 SR+HIP vs M2 SR+HIP+HT. 
Table 6 tabulates the properties which were determined to be different (at the 0.05 level) 
for each statistical metric in these comparisons. 


Table 6 Statistically Significant Differences (p<.05 ) Detected in Group Comparisons 


Comparison 

Means 

Variance 

Median 

Distribution 

Ml SR vs Ml 
SR+HIP 

Gu, Gy, E 

Gu, Ar 

Gu, Gy, E 

Gu, Gy, E 

Ml SR+HIP vs 
Ml 

SR+HIP+HT 

Gu, Gy, Hr 

Gy, € 

Gu, Gy, Hr 

Gu, Gy, Hr 

Ml SR vs Ml 
SR+HIP+HT 

Gu, Gy, E, Ar 

Gy, E, e 

Gu, Gy, E, Ar 

Gu, Gy, E, Ar 

M2 SR vs M2 
SR+HIP 

Gu, Gy, E, 6, Ar 

E 

Gu, Gy, E, 6, Ar 

Gu, Gy, E, e, Ar 

M2 SR+HIP vs 
M2 

SR+HIP+HT 

Gu, Gy, 6 

Gu, , Gy, 6, Ar 

Gu, Gy, 6 

Gu, Gy, e 

M2 SR vs M2 
SR+HIP+HT 

Gu, Gy, E, 6, Ar 

Gu, Gy 

Gu, Gy, E, 6, Ar 

Gu, Gy, E, e, Ar 
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What degree of property improvement do the HIP and heat treatment steps impart? Table 7 
summarizes the difference in each of the mean material properties for the comparison (Table 6 
indicates whether these changes are statistically significant). 


Table 7 Summary of Percent Difference in Mean Material Properties for Group 
Comparisons 


Comparison 

Ou 

a y 

E 

e 

Ar 

Ml SR vs Ml 
SR+HIP 

11.0 

3.3 

28.3 

8.7 

-0.4 

Ml SR+HIP 
vs Ml 

SR+HIP+HT 

9.2 

25.3 

1.25 

7.6 

42.2 

Ml SR vs Ml 
SR+HIP+HT 

21.1 

29.5 

29.9 

17.0 

41.5 

M2 SR vs M2 
SR+HIP 

8.25 

3.7 

7.6 

-7.9 

-11.4 

M2 SR+HIP 
vs M2 

SR+HIP+HT 

8.3 

25.0 

-1.5 

-9.2 

1.7 

M2 SR vs M2 
SR+HIP+HT 

17.3 

29.6 

5.6 

-16.4 

-9.9 


If we look at successive plots of the data distributions as the samples go through HIP and heat 
treatment, we can clearly see the observed trend of strength property improvement with 
additional processing. As an example, Figures 6 and 7 display a scatterplot and a box and 
whisker plot, respectively, of ultimate tensile stress data for the groups Ml SR, Ml SR+HIP, and 
Ml SR+HIP+HT. This property improvement also extends to yield strength and elastic 
modulus. A similar trend is observed for the analogous M2 data. 
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Scatterplot by Sample 
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Figure 6 Scatterplot comparing ultimate tensile stress data (in KSI) for Ml SR, Ml SR+HIP, 
and Ml SR+HIP+HT. 


Box-and-Whisker Plot 
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Figure 7 Box and whisker plot comparing ultimate tensile stress data (KSI) for Ml SR, Ml 
SR+HIP, and Ml SR+HIP+HT. 
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Metallurgical characterization of samples built on the same machine but at different 
stages in the process (i.e. as-built, HIP, and HIP + subsequent heat treatment) is essential to shed 
light on the microstructural evolution which imparts the observed statistically significant changes 
in properties. HIP can reduce or eliminate the presence of pores - this improved material 
consolidation may contribute to enhanced mechanical perfonnance, as HIP is associated with a 
large increase in yield strength for the machine variability data. Material changes which take 
place during HT harden the material and seem to increase the ultimate tensile strength. This 
hardening may have an ancillary effect of making the specimen more brittle, reducing the 
percent elongation values (although this is not the case for the M2 data). The broad conclusion 
from this comparison is that HIP does more to improve yield strength, while HT has a more 
significant impact on ultimate tensile strength. 

There are some anomalous results which could potentially be explained by metallurgical 
evaluation: 

a) The large increase in modulus with additional processing which was observed for the Ml 
specimens but not for M2 

b) Why percent elongation and area reduction increase as the Ml specimens go through HIP 
and HT, but decrease for the M2 specimens 

c) For identical specimens built at the same parameters but on different machines, 
postprocessing can narrow the initial property differences observed in the as-built 
samples to a level that is no longer statistically significant. This is apparent in Table 8, 
which indicates the differences in properties which are significant at the .05 level for the 
Ml SR/M2 SR, Ml SR+HIP/M2 SR+HIP, and Ml SR+HIP+HT/M2 SR+HIP+HT 
comparisons. While the as-built (SR) specimens are characterized by significant 
differences across properties (c u , c y , E, e, A r ) and statistical metrics (mean, variance, 
median, distribution), the specimens which have gone through HIP and heat treatment do 
not exhibit these differences. 
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Table 8 Statistically Significant Differences (p<.05 ) Detected in Group Comparisons 


Comparison 

Means 

Variance 

Median 

Distribution 

Ml SR vs M2 
SR 

Gu, Gy, E, £, Ar 

Gu, £, Ar 

Ou, Oy, E, 6, Ar 

Ou, Oy, E, e, Ar 

Ml SR+HIP vs 
M2 SR+HIP 

Ctu, Oy, 6, Ar 

Gu, £, Ar 

Oil, Oy, 6, Ar 

Ou, Oy, 6, Ar 

Ml 

SR+HIP+HT vs 
M2 

SR+HIP+HT 


Ar 




Table 9 catalogs the percent difference in the mean material properties for the data groups 
being compared. It is apparent that the difference in mean material properties is reduced 
by the HIP process and further reduced by heat treatment. While the ultimate tensile 
strengths and yield strengths for the Ml SR and M2 SR specimens initially have the 
smallest discrepancy, these differences were found to be significant at the 0.05 level (see 
the corresponding entry in Table 8). The large disparity in modulus, elongation, and area 
reduction “shrinks” with HIP; following heat treatment, mean differences in mechanical 
material properties for the Ml and M2 specimens were not found to be significant. 


Table 9 Summary of Percent Difference in Mean Material Properties for Group 
Comparisons 


Comparison 

Ou 

Oy 

E 

e 

Ar 

Ml SR vs M2 
SR 

4.3 

1.0 

22.0 

45.4 

66.8 

Ml SR+HIP 
vs M2 
SR+HIP 

1.8 

1.3 

2.6 

23.2 

48.4 

Ml 

SR+HIP+HT 
vs M2 

SR+HIP+HT 

1.0 

1.1 

- 0 . 13 * 

3.9 

6.1 


*negative value means that M2 group mean is greater 
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The equalization effect of postprocessing and heat treatment in reducing machine variability can 
be visualized using box and whisker plots. As an example, we examine the evolution of yield 
strength for specimens built on the M 1 and M2 machines as they undergo HIP and heat 
treatment. Figure 8 shows the box and whisker plot for the yield strengths of Ml SR, M2 SR, 
Ml SR+HIP, M2 SR+HIP, Ml SR+HIP+HT, and M2 SR+HIP+HT. Mean yield strength 
improves for both Ml and M2 specimens with additional processing and the amount of overlap 
between the M 1 and M2 data sets decreases to a point that the differences between the data sets 
are no longer statistically significant. 


Box-and-Whisker Plot 


Ml, SR 
M2, SR 
Ml, SR+HIP 
M2, SR+HIP 
Ml, SR+HIP+HT 
M2, SR+HIP+HT 



120 130 140 150 160 170 180 

Yield strength 


Figure 8 Comparison of means for yield strength (KSI) 
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It is clear from the two-group analyses that HIP and heat treatment are necessary not only to 
improve properties (specifically to elevate the properties of additively manufactured materials to 
values that resemble their cast or wrought counterparts), but also to reduce the effects of process 
variability. This observation is confirmed by the ANOVA summarized in the next section. 


Multigroup Analyses 

Comparison of multiple groups were performed using the ANOVA (analysis of variance) 
technique to test the (null) hypothesis of equal population means. Thus rejection of the null 
hypothesis (which occurs when p<0.05 ) in this context indicates that the samples originate from 
populations whose means are significantly different. 

While the results of the ANOVA are (and ideally should be) a confirmation of the two group 
analyses, it does not render the latter unnecessary. The danger in making comparisons across 
many pairs of means is that the probability of an error on at least one pair increases, which is 
why it is important to also perform the two variable comparisons previously summarized in 
addition to the ANOVA. 

Material properties for the following data groupings were compared using ANOVA: Ml SR, M2 
SR, Ml SR+HIP, M2 SR+HIP, Ml SR+HIP+HT, and M2 SR+HIP+HT. The analysis considers 
the combined effect of machine variability and differences in post-processing/heat treatment. 

This comparison represents the most succinct and compelling summary of the variability data, as 
they show simultaneously how properties vary with equipment and postprocessing conditions. 

Examining the data among the six groups simultaneously, we note several key trends: 

• The effect of HIP and heat treatment at reducing the effect of machine variability on 
material properties is clearly illustrated by the means and quantile plots. The means plots 
for ultimate tensile strength, yield strength, and modulus show that the amount of overlap 
between the M 1 and M2 sample groups increases with HIP and that the intervals which 
bound the means are nearly coincident for the Ml SR+HIP+HT and M2 SR+HIP+HT 
data sets. This equalization effect was noted previously in the two variable comparative 
analyses, but is confirmed further by the ANOVA. 

As an example of this, the means plots for ultimate tensile strength for the multi-group 
comparison is displayed in Figure 9. The bars which bound the mean (denoted by a 
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circle) represent the uncertainty intervals. The amount of overlap between the group 
means for the Ml and M2 groups increases with additional processing. 


Means and 95.0 Percent LSD Intervals 
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Figure 9 Means plot for ultimate tensile stress (KSI) 


• The distance between the cumulative probability distributions for the data sets Ml and 
M2 “shrink” with HIP and decrease further with heat treatment, which can be seen on a 
quantile plot. This graph plots the proportion of data in each sample that is below a given 
value of X, as a function of X. The function is cumulative (ex. the x value corresponding 
to a value of 1 is the upper limit of the data since 100% of the samples fall below it). 
Quantile plots which lie very close together indicate that the samples come from the same 
population/distribution, while an offset of one plot relative to another indicates that the 
distributions are different. The Kolomogorov-Smimoff test, which characterizes the 
statistical significance of the maximum distance between quantiles of the data groups 
being compared, is visually represented by these plots. The quantile plot for mean 
ultimate tensile strength is shown in Figure 10. Statistically, this narrowing of the gap 
between lines on the quantile plot with additional processing (HIP+HT) means it is more 
likely that the data sets being compared originate from the same population distribution 
(a visual observation confirmed by the results of the Kolmogorov-Smirnoff test). 
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Quantile Plot 
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Figure 10 Quantile plot for mean ultimate tensile stress (KSI) 

• Trends in changes in percent elongation and area reduction with heat treatment and 

machine are less apparent (although the differences in means among the 6 groups are still 
significant at p=0.05). There is substantial vertical overlap for these properties among 
data groups in the means plot (Figure 1 1 shows the percent elongation plot as an 
example). The quantile plot is characterized by crossover (which indicates a failure to 
accept the K-S alternative hypothesis that the population distributions for the sample sets 
are significantly different). Figure 12 shows the quantile plot for percent area reduction 
among the six groups. 
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Means and 95.0 Percent LSD Intervals 



Figure 1 1 Means plot of percent fracture elongation 
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Figure 12 Quantile plot of percent area reduction 


Statement A: Approved for public release; distribution is unlimited. 

DESTRUCTION NOTICE - For classified documents, follow the procedures in DoD 5200. 22-M, National 
Industrial Security Program Manual, Chapter 5, Section 7. For unclassified, limited documents, destroy by any 
method that will prevent disclosure of contents or reconstruction of the document. 


Conclusions and Future Work 


The key conclusions of the study presented here are: 

1) It is possible of identical samples which underwent the same heat treatment but were 
produced on different machines to exhibit a statistically significant difference in 
characteristic mechanical material properties. 

2) In general, there is a statistically significant difference in characteristic mechanical 
material properties for samples which were produced on the same machine but underwent 
different heat treatments/postprocessing regimes. This difference usually represents a 
property improvement. 

3) While the purpose of HIP and heat treatment is to improve mechanical properties by 
reducing porosity and increasing strength, the postprocessing regime also has the 
ancillary benefit of eroding the impact of machine variability on properties to a level that, 
in the case of the specimens which underwent both HIP and heat treatment, its impact on 
group means is no longer statistically significant. 


As noted in the two group analyses discussion, a follow-on investigation of the specimens in 
the machine variability study should include optical microscopy of sample cross-sections and/or 
SEM analysis. Microscopic evaluation will provide insight into what specific changes in the 
microstructure brought about by additional processing are responsible for the a) the equalization 
effect (where variability in mechanical properties for the M 1 and M2 groups is reduced by HIP 
and further reduced by heat treatment). Microscopy can also potentially explain some of the 
anomalous/nonintuitive trends in the data (such as why a large increase in modulus with 
additional processing was observed for the M 1 specimens but not for M2 or why percent 
elongation and area reduction increase as the Ml specimens go through HIP and HT, but 
decrease for the M2 specimens). 

A second machine variability study is currently planned to characterize how machine 
variability impacts mechanical perfonnance in environments that are of interest to the propulsion 
community. This study will include an addition 60 samples that will be tested in liquid nitrogen, 
liquid hydrogen, and at both elevated and cryogenic temperatures characteristic of operational 
propulsion systems. 

A round robin study for machine variability should also be a priority for further research 
activities. As AM techniques are increasingly adopted by government and industry to produce 
propulsion hardware in human-rated systems, development work is needed to establish these 
materials as reliable alternatives to conventional subtractive manufacturing. Machine variability, 
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based on this and other studies, has been recognized as a factor that influences properties of 
additively manufactured parts and that baseline material properties might not be maintained 
when parts are produced on different machines. For future programs, NASA will procure parts 
from vendors who use a diverse range of machines to produce parts and it is essential that NASA 
(as well as the broader AM community) develop a sound understanding of degree to which 
machine variability influences material properties. A round robin study would characterize 
variability between machines used by vendors and NASA MSFC AM laboratory. In this class of 
study, vendors, original equipment manufacturers (OEMs) of AM machines, and NASA would 
produce parts of the same material and geometry at the same parameters, but on a wide range of 
machines. Such work is also needed to assess whether machine variability effects are more 
pronounced in parts built using powder bed fusion (PBF) techniques, where implementation of 
robust process controls is perhaps more difficult than for wire fed systems. 

In summary, the aerospace community faces unique challenges that must be addressed 
before the potential design and economic benefits of AM can be realized. Currently AM parts 
are process specific and standards are required to assess sensitivity of parts to base material 
(powder) composition, address machine variability, establish post-processing controls (i.e. heat 
treatment and surface treatment protocols), and ultimately ensure that AM products are safe and 
reliable prior to their deployment in crewed systems. As we have demonstrated, even with robust 
process controls, it is possible for two AM machines operating at identical parameters with the 
same base material to produce parts with slightly different material properties. While 
understanding machine variability is only one link of many in the chain to establishing “certify 
as you build” process robustness, its characterization is critical to design, analysis, hardware 
acceptance and certification of AM parts. Though AM technologies will not supplant 
manufacturing for high rate production, low cost parts anytime soon, it is attractive for low rate 
complex geometries that are characteristic of rocket propulsion hardware. To leverage AM’s 
promise for creating affordable propulsion systems, we must undertake the foundational work 
necessary to understand the process and mitigate risk associated with replacing conventional 
subtractive manufacturing techniques with AM in human rated aerospace systems. 
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